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ABSTRACT 



The Global 2000 project was a 3-year project aimed at 
improving literacy in five Massachusetts manufacturing companies. At four of 
the five Global 2000 program sites, an external evaluator conducted focus 
groups and open-ended interviews with members of the projects' employee 
involvement teams, previous students, and teachers. The evaluator also 
reviewed the needs assessment and evaluation documents developed and used at 
each site. The evaluation established that the classes taught by the 
educational provider were strong. All sites had partnerships among key 
stakeholders and a plan and goals statement; however, the extent to which all 
stakeholders contribute to project goals and activities varied significantly 
from site to site. The greatest weakness of Global 2000 appeared to be in the 
area of collecting and analyzing data. It was recommended that the data 
collection instruments of one of the business partners (Fire Controls) be 
adopted as a model for the other sites. Only at one site were reviews of the 
program and its employee involvement team mixed. (Appendixes constituting 
approximately 50% of this report contain the following: draft focus group 
protocol; teacher protocol; description of the 12 -step evaluation process; 
goal-setting worksheets; and guidelines for curriculum documentation.) (MN) 
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INTRODUCTION 

At the request of Lloyd David and Katherine Archer, I conducted an 
evaluation of programs at four sites which are part of CEI's Global 2000 
Program. I was not brought on as the external evaluator required by the 
National Workplace Literacy Program to evaluate CEI and its process in 
providing workplace education, but more as a local evaluator with an 
expertise in qualitative methods to focus on the local programs themselves. 
This evaluation is therefore only one piece of the evaluation work CEI is 
doing. The purpose of this evaluation is to improve the programs offered 
at these sites. In addition, the information gathered will enable CEI to 
move ahead with the task of further developing an evaluation component to 
their program development and goals-setting process. The questions 
guiding this research were: what was good and what could have been 
better about the previous year, and how can we improve services? The 
specific objectives were: 

• to determine what teachers, students, and company personnel liked 
(strengths) and disliked (weaknesses) about the program 

• to determine if the programs achieved their goals 

• to determine whether additional goals or expectations have emerged 



This report is divided into four sections. The first section describes the 
process used to gather information about 1) participants' perceptions of 
the ways in which the program has been successful so far and the ways in 
which services and the program can be improved, and 2) achievement of 
goals and extent of progress. The second section provides 
recommendations regarding documentation and evaluation tools developed 
and used by CEI. The third section provides site-specific analyses. And 
finally, the fourth section provides an analysis of CEI's Global 2000 
project from the perspective of teachers in both the ESL and ADP 
programs, concerned primarily with communication within the program, 
curriculum, and assessment issues. 
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SECTION 1: PROCESS and METHODOLOGY 

I conducted focus groups and interviews with members of employee 
involvement teams, with previous students, and with teachers for four sites 
in the Global 2000 project 1 . I prefer focus groups and interviews because 
in order to get deep information it is necessary to ask open-ended questions 
which people often do not answer on surveys. Also, in focus groups, a 
comment made by one person often triggers a comment by another person. 
Further, it provides me with the opportunity, not found in surveys, to ask 
.clarification questions and confirm my understanding. The choice of focus 
groups or interviews fell to the contact person at each site; while I mainly 
used focus groups, in a couple of places they felt that interviews would be 
easier jn terms of taking people off the production floor. At one site. 
Power General, they could not arrange a meeting time, so they asked me to 
send my interview questions; EIT members filled them out and returned 
them. At this site I also spoke with students one-on-one. I asked similar 
questions of both students and EIT members. Refer to the appendix for the 
interview protocols. After conducting all the interviews and focus groups, 

I reviewed the data to identify common themes. 

In addition to speaking with students and EIT members, I interviewed both 
ESL and ADP instructors. I also spoke with Lloyd and Katherine early in 
the process, and with Kathy midway through. 

Finally, I reviewed the needs assessment and evaluation documents 
developed and used at each site, and the information gathered through these 
documents. I reviewed these not to determine the quantitative results, 
which CEI is able to do through NWLIS and its own charts, but in order to 
see comments and to note the format or structure of these documents. 



1 . The sites included in the evaluation arc Analog Devices, Fire Control, Power General, and Boston 
Scientific; AEC did not participate in the evaluation. 
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DIFFICULTY 


RECOMMENDATION 


For some sites, there were student 
evaluations but no teacher evaluations, or 
vice versa (e.g. FCI). Therefore, it is not 
possible to compare responses; this kind of 
comparison (triangulation) is used in 
qualitative research as a way to establish 
validity. 


Ensure that teachers and students fill out 
similar forms. 

To make comparisons among different 
respondent groups simple, keep items in the 
same order on different forms. I.e. the list 
of items on a needs survey should be in the 
same order as the list on the teacher's 
evaluation, the same as on the student's 
evaluation. If teachers respond to items that 
students do not (or vice versa), put those at 
the end of the list. 

Data collection forms should match the 
objectives to be evaluated and each other. 
Each item on an evaluation form should 
match an indicator or progress or 
achievement for the objectives. 

Make sure each item only asks one thing 
(avoid and/or statements because then the 
respondent must answer two questions in 
one - see supervisor questionnaire at Power 
General) 


Many forms were not dated. Therefore I 
could not tell which cycle they referred to. 


Date all forms. Include site name (fully 
written), cycle, class level, teacher's name. 

Each form should have a clear purpose. 
Give each form a name to distinguish it 
from others. Not all forms should be called 
evaluation forms; for example, rather than 
"ongoing evaluation", try "class log" or 
"assessment scores". It may help to 
distinguish between assessment and 
evaluation. According to the Department of 
Education, assessment refers to the process 
of collecting and analyzing information on 
student learning while evaluation refers to 
the process of collecting and analyzing 
information on a variety of aspects of a 
orogram (including assessment of learning, 
transfer of learning, organizational change, 
orogram processes and outcomes) to 
determine its value. 
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Forms did not match class lists (or there 
were no class lists). Therefore I do not 
know which cycles and classes they are for 
and if there are forms for all classes held. 



Goals statements indicate company goals 
but do not indicate the role the Global 2000 
project plays in addressing those goals, 
l.e., while a company goal may be to 
"increase operator's abilities to read and 
understand operating procedures" (FC1), if 
no operators are enrolled in classes, the 
Global 2000 project should not be evaluated 
on whether or not operators improve their 
ability to read and understand procedures. 
Or, if a company goal is to increase 
productivity, there is no statement on the 
relationship between the basic skills 
program and this goal (how will improved 
basic skills help the company's bottom 
line?). 



Collect data on clearly defined and 
articulated objectives. It may not be 
possible to collect data on all goals. 

If a company's goal is to increase 
productivity, the goals statement should 
indicate in what ways the Global 2000 
project fits with that goal, and it should be 
evaluated on how it performs in those 
ways, not on improved productivity (there 
are other factors that contribute to 
productivity). 

The objective being evaluated should have a 
minimum of cause-effect relationships. 

I.e., we believe that when workers improve 
their reading skills, they can read and 
understand work orders/procedures. When 
they understand them, they make fewer 
errors. When workers make fewer errors, 
there is less waste. When there is less 
waste, productivity increases. (Note that 
this assumes that the workers in question 
are the ones who read the orders. It also 
assumes that there are no other factors 
impinging on whether or not they actually 
read something they know how to read.) 

Some companies keep statistics on errors 
and waste. Global 2000 keeps records on 
reading comprehension skills. As reading 
skills go up, do errors and waste of class 
participants go down? What else might 
contribute to errors and waste? And how is 
the company addressing those? The 
question therefore is not "What are the 
company's goals?", but "What are the goals 
for this project?" 

Comparative data is crucial to determining 
the impact of the program. 

FCI has collected this kind of data - perhaps 
they could present their methods to other 
sites. 



ERIC 
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Rating scales varied from form to form 
(e.g. evaluations at ADI had a different 
scale from the needs survey, so scores 
could not be compared). 


Use the same rating scale on all forms. 
Rating scales should be even in number. It 
is best on each form to ask the respondent 
to rate current level of ability rather than to 
rate progress; by comparing pre and post 
survey scores, the evaluator can then 
determine degree of progress. If numbers 
go down, don't immediately assume failure; 
rather, look for possible reasons. For 
example, as students improve their skills, 
they become more aware of how little they 
know and of how many mistakes they 
make. They may therefore rate their skill at 
a lower level. It's not that their skill 
decreased but that their awareness 
increased. When you find apparent 
anomalies, talk to people to find out why. 
(another area of anomaly is students rating 
progress less than teachers do) 

Items to be rated should be written as 
statements (grammar and punctuation) 

Make sure that items to be rated can be rated 
with that scale rather than requiring a yes/no 
response. 
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SECTION 3: SITE-SPECIFIC EVALUATIONS 

Analog Devices, Inc. 

Strengths (Likes) and Weaknesses (Dislikes) 

Based on the information obtained in student and team focus groups, the 
program at this site is well-organized. The team is strong and both team 
members and students were positive about the results so far. The existence 
of the team is seen as a strength and a sign of success. Also, students do 
homework, which indicates commitment; students also act as examples to 
others. Additional "likes" include the time of classes and the opportunity to 
use what they learn. 

The primary concern expressed by the team was that they lacked the tools 
to assess people at the beginning, so they will never know the full benefit of 
the program. In this, managers expressed disappointment with CEI; they 
expected CEI to be better prepared to do initial assessments. Also, they 
find that since the learning curve is slow, it is difficult to assess progress. 
An additional concern is that the ESL teachers are not as strong as the ADP 
teachers. Apparently, students felt that the teacher was too mild and 
relaxed; they prefer someone who is more strict. They had high 
expectations which were not met; they did not feel challenged enough. On 
the other hand, because it was a multi-level class, other students were 
uncomfortable because it was too difficult. The team recommended that 
more/better testing be done initially to place people in the right class and 
that no one be allowed in without having been tested first (apparently some 
people got in without being tested). 

Goal Achievement 

The primary goals they identified were to get everyone in the company to 
read and understand job procedures and notes, and to do it right without 
relying on translators. To know whether they are achieving these goals, 
group leaders pay attention to workers' use of reading, writing, and 
speaking. What group leaders and team members have noticed is that "a 
wall came down"; that is, people from different language backgrounds 
speak English with each other, as do people from the same language 
background (rather than reverting to their native language). They sense an 
increased comfort level; it has become the atmosphere in the company to 
speak English. In addition, workers now read procedures and ask for help. 
They know people are reading because of the kinds of questions they ask 
and because they point out errors. As one student said, "I know my 
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progress because I know my own problems. So I can see change. Now I 
learned a little so I have a million questions; 1 notice things and want to 
know." These comments demonstrate that the goal of improved 
communication skills (from ADI's goal document) is being achieved. 

From a statistical analysis of the student and teacher evaluations, I was able 
to determine that teachers tend to rate greater progress than the students. I 
was not able to determine why that is. 

There was no information available on other goals indicated on the goal 
document, including improved productivity/employee empowerment, 
improved team problem solving, and improved process control. We need 
more information on the link between improved communication and these 
larger goals. Also, I am not sure what the connection is between 
productivity and empowerment (they are linked in one goal). The goal 
document should clearly spell out how the measures work (e.g., should the 
number of ECNs generated go up or down? ) 

Boston Scientific 

Strengths (Likes) and Weaknesses (Dislikes) 

The team is a strength of this program as well. The company finds that the 
model of targeting one or two departments at a time is a strength because 
participants get support from supervisors and co-workers while they are in 
class; the whole department buys into having them participate. The only 
dislike was that expressed by students: 15 weeks are not enough. The EIT 
needs to explore ways to continue supporting student learning and practice 
after classes are over. One possibility is mentoring/tutoring, as is done at 
Fire Controls. Another possibility is to set up a drop-in learning center or 
library where students can go on their own time. Students liked the fast 
pace of the classes ("no time wasted") and the talking. There were mixed 
feelings about the textbooks. 

Goal Achievement 

The goals of the program, as articulated in the team focus group, are to 
improve the literacy skills of employees and to help them be more 
participative. It is understood (or believed) that these will help the 
company. It is also understood that with only 15 weeks of instruction, 
students will not gain fluency; the team views the program as a building 
block. Specifically, they want increased confidence, for students to try to 
use English more, even after the class is over. (I did not have a goals sheet 
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document for Boston Scientific). In terms of improved literacy skills, 
according to team members, there is noticeable progress. Workers are 
able to point out errors in written procedures, which demonstrates that 
they are able to, and do, read. They also make suggestions for 
improvement. People involved in the program have been promoted, taken 
on more responsibility, and received company recognition awards. Those 
who have participated in the program will now leave notes for each other, 
which they did not do before. Note however, that the team and the 
company do not keep any records on these achievements. From needs 
survey and evaluation documents (which were not dated, so I am not sure 
when they refer to and which go with which), it appears that participants' 
ability to read and follow directions and writing of notes increased only 
slightly; this is not surprising given that literacy skills take a long time to 
develop (according to the DOE, to move from SPL level 0 to level 1 takes 
over 100 hours of instruction), but it does seem contrary to focus group 
information. The greatest increases (in the documentation) were in 
comfort level and in understanding verbal instructions. The increased 
participation in team meetings in the company is an indicator of progress 
for the program because students learn to speak up, do presentations, and 
to function as a team in class. 

From the students' point view, the fact that classes run for a short period of 
time means that they must be very focused and that no time can be wasted. 
They found that in this short time, they do become more confident and they 
are able to better comprehend speech. However, they do not feel that they 
make great progress in terms of grammar, vocabulary, and writing. 

Additional Goals 

The team would like to mold the program more to the company. For 
example, they mentioned using Standard Operating Procedures as a tool for 
teaching English because it is important for operators to read them. But, 
they also mentioned that these procedures are always changing, which 
means that it is not necessary to teach the procedures themselves in class. 

In fact, if the procedures are always changing, then what is important is to 
develop transferable reading skills (regardless of what document/text is 
being read). The team also mentioned continued focus on "team", and on 
managing class in such a way as to promote the team concept (what it is, 
how it operates, the roles people play). 
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Power General/Nidec 



Strengths (Likes) and Weaknesses (Dislikes) 

At Power General there were mixed reviews about the program and about 
the Employee Involvement Team. Some found the team effective in its 
responsibilities while others felt it was not so effective. Those who 
considered it a strength did so because it includes people from all levels and 
functional areas within the company, meaning that all needs are considered 
and represented in goal-setting and development. Others articulated 
numerous challenges or weaknesses, including a lack of regularly scheduled 
meetings, a great difference of opinions making it hard to come to 
decisions, and the lack of English fluency among all members. In addition, 
the team is challenged by the responsibility of measuring changes in work 
productivity. In general, there was agreement that the team was effective 
in encouraging workers to sign up for class. 

One person stated that it seemed like all decisions were made in advance by 
others so there was no real point in her participation. For example, the 
process of selecting participants seemed particular and pre-determined 
(even though recruitment was open); she did not feel that she had any 
input. Also, she was not clear on what was expected of the team in terms 
of setting goals and creating the project. Although only one person openly 
stated such confusion and dissatisfaction, it is possible that others feel the 
same way but were not open in stating it. One student and EIT member 
also stated that the team was good because everyone gave ideas, but when 
pressed also admitted that she never spoke up because "my idea is low". If 
a few members give the impression of having all the answers and decisions, 
then others may only agree or be silent. In addition, team members 
articulated numerous, different responsibilities; this shows that they are 
either unclear on what their responsibilities are (and therefore have 
different perceptions) or they may be overwhelmed by having too many. 
(Note: I did not meet with the team as a focus group because they could 
not all meet with me at the same time; instead, they answered open ended 
questions on a form). 

Another strength of the program was the teachers. 

Goal Achievement 

Despite the mixed reviews on the functioning of the team and on evidence 
of changes in productivity, respondents were positive about the programs 
affect on participants' language skills. One supervisor wrote that for her, 
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evidence of progress was that. a few people on her line now sometimes 
disagree with her; they have the "guts" to do so; in addition, many team 
members and students said that they ask more questions than before. Also, 
after the classes they were able to read the bill of materials and point out 
errors. Others also pointed to increased communication of problems or 
errors, increased adaptability, and improved job performance. Other 
evidence of progress has been that participants no longer use interpreters, 
they communicate with each other, write notes to the second shift, and 
check documents before starting a new job. Also, there have been changes 
in terms of confidence to communicate. They felt that they had many 
examples of individual/anecdotal successes and perceptions of 
improvement, but "no clear way to measure accurately", no concrete 
effects to the business' bottom line; in other words, they believe that data 
must be quantifiable to have value. 

Additional Goals 

To continue on as they have been, but to get clearer evidence of goal 
attainment. One person felt that the team needed more direction and 
research from CEI regarding recording results of the program. 

Fire Control. Inc. 

Strengths (Likes) and Weaknesses (Dislikes) 

The EIT is a great strength of this program because those on the team 
(except for one) all work directly with program participants. They are 
very organized, which demonstrates their commitment to participating in 
the process. The team implemented two components which are also 
strengths of the program: tutoring/mentoring, and pre/post surveys. 

The tutoring piece provides students with someone to go to if they have 
questions or want to practice something. The problem with the tutoring 
program is that students are uncomfortable going to see their tutor because 
they feel like they are bothering them; also, sometimes, by the time they 
manage to track the tutor down, their break is over. The team would also 
like to see managers and engineers become tutors (rather than just team 
members). The concept of the tutor is good because it provides a way for 
students to practice outside of class; to ensure that students are comfortable 
with it, it may need to be formalized more in terms of time. 
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Team members across the board were very positive about the results of the 
program. Students also were positive. They would only have preferred 
greater strictness on the part of the teacher. 

Goals Achievement 

The team identified several goals for the program. These included: 
raising the English comprehension and speaking levels of workers 
increasing workers' understanding of what they are building 
improving efficiency 

improving comprehension of visual aids and documentation 
decreasing workers' fears of asking 

moving workers forward so they become a resource of the company (not 
just people who use resources) 
improving the quality of their products 

All members in the focus group agreed that they had achieved more than 
they had expected because they had understood it would be a long-term 
process; now they are more aware of what a program like this can 
accomplish. Students also felt that they had improved their skills, and have 
an opportunity to use those skills, but also that they forget a lot when no 
longer in classes. The team has achieved clarity on the help their workers 
need. They find that those workers who have participated in classes speak 
now more than before. They ask more questions and the questions are of a 
different kind, demonstrating critical thinking. They got over a certain 
level of shyness (a barrier was broken); participants feel more like a part 
of the organization, and they have become more comfortable with the 
program itself. Also, the team has documented improvements (through 
statistical analysis) in quality of work as a result of program participation. 
This latter achievement is a key one because it is lacking in most programs. 
It demonstrates forethought in terms of what and how to collect data; other 
programs would benefit from hearing what FCI did from the beginning 
(when they did not yet know that they would have a lab-type setting). I 
highly recommend that you document and disseminate this information in a 
journal article. 

Additional Goals 

The EIT expressed an interest in meeting with other companies who have 
programs, to share ideas. They also expressed an interest in further 
tailoring the program to their own terms and processes after funding is 
over (and not worry about the difference between education and training). 
An additional goal is to improve the tutoring component. 
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SECTION 4: TEACHERS' PERSPECTIVES 

ADP Program 

Overall, ADP teachers were positive about the program. In particular, 
they feel that Kathy Hassey is doing a very good job; specific comments 
were that she provided structure to staff meetings, that she is working hard 
on curriculum and assessment issues, and that she goes the extra mile. 

Areas of concern primarily revolve around perceived lack of clarity in 
communication. 

Strengths of the program include 

• demands made of students are rigorous 

• students are motivated and eager 

• the mission of having students get their diploma; the vision of helping 
educationally disadvantaged people 

• staff meetings which were helpful in sharpening the focus on key issues, 
especially that of what students need to do in order to graduate 
legitimately/extent of contact between/among teachers was sufficient 

• organization of program in three strands 

• extent of class time 

• individualized attention students receive because of small class sizes 
(note that some teachers felt classes were too big: agreement was that 10 
was an optimal number) 

• supportive atmosphere (in supporting students) 

• commitment of teachers 

• academic (rather than experiential) approach of program (noted as a 
strength by one instructor) 

Weaknesses/areas for improvement 

• responsibilities of instructors are vague: counseling seems to be part of 
the job description, but it is not clearly defined (seems to include 
tutoring, phone calls, meetings). A lot is asked of teachers; as a result, 
committed teachers put in extra time for which they are not paid. 

• curriculum is vague (but note that this is partly due to the high turnover 
recently in the Coordinator position, and work is now being done in this 
area, which staff recognized): there needs to be a concerted effort to 
draw up and establish an integrated curriculum, to develop or find 
materials suited to this population (especially reading materials); it is a 
problem that there are limited materials which are appropriate to this 
population 
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• methods/approach to use in assessing students has been vague; skills 
necessary for receiving the diploma, considering that they have such 
little time and many students cannot write at all (system and criteria 
were changed in the middle of a year, which was disconcerting to 
students) (note that some teachers indicated that clarity was needed on 
what students need to do in order to graduate legitimately, while others 
felt that clarity had been reached) - it may help to develop benchmarks 

• suddenness of communications has been problematic (see above 
example) 

• a written mission statement or statement of purpose would be 
appreciated 

• communication needs to be improved (and written): mission, grading 
system, expectations for graduation, curriculum 

• visitors to classes were found to be an interruption, particularly when 
there was little or no warning; it was felt that visitors should be 
observers but not participants because the time is so limited and the 
amount they need to cover is so great 

• goals and objectives need to be more clearly defined; there was felt to 
be a lack of consistency 

Teachers' understandings of program goals 

• to bring students to the point where they can graduate from high 
school/get a high school diploma 

• (given this goal, teachers need a clear understanding of what is expected 
of students for them to be able to do this) 

• to enhance people's personal development and the company's pay-off in 
productivity (hence a customized curriculum; the greatest connection 
between getting the diploma and personal development was improved 
verbal and written communication) 

• to make employees more confident and self-assured so they function 
better in the workplace and in society 

Given the focus on a high school diploma, the teachers did not see any 

importance to being in a workplace environment; the environment and the 

funding source had no affect on their teaching. 

Evidence of goal attainment 

• progress in verbal communication and confidence to present to others 
(confidence means willingness to speak, to not be quiet or timid, to 
stand up in front of a group and talk; this was evident in the 
presentations at Boston Scientific) 
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• students get their diplomas (but no knowledge of long-term effects) 

• use of exams, reports, essays, and presentations: measured against the 
quality of their work in these areas at the start of the class 

ESL Program 

Strengths of the program 

• flexibility of program to respond to needs of a particular class 

• staff meetings; extent of contact with other teachers (teachers meetings 
were run well in that Katherine helped them listen to each other and 
reflect on their work); good exchange and sharing of ideas 

• fact that Katherine took care of issues with management so that teachers 
were free to just teach (note that there was disagreement on whether this 
was a strength or a weakness) 

• tutoring program at Fire Controls 

• Teams: a strength because was very active (but also a weakness - see 
below) 

Weaknesses/areas for improvement 

• a curriculum guide is needed 

• a written list of objectives (both this and the curriculum guide would 
help give the teacher direction) 

• the fact that the instructor is not involved in the team was seen by one 
teacher as a weakness because she felt that there was a communication 
breakdown in the dispersal of information (people hear differently and 
report selectively, so when one does not get the information first hand, 
one may be missing something of importance); also, since this teacher 
has a lot of experience with a team-based approach, she felt that she 
could have provided valuable input 

• classroom space (such as a conference room rather than the cafeteria) 

• materials: an on-site "library" for students; materials which apply skills 
in more ways (moving students to apply skills beyond the classroom) 

• most teachers on the ESL staff come from backgrounds in reading and 
learning disabilities (two very important areas) and with children. They 
do more reading/writing/phonics activities with ESL books and less 
work on speaking/listening/conversation. For some teachers, the 
transition from children to adults and from school environment to 
workplace environment has been a difficult one (but Katherine has done 
a good job of orienting people to teaching in this environment) 

• it may help to develop benchmarks or some other "standard" way to 
indicate student progress so it is clear on forms what the marks mean 
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Teachers' understandings of program goals 

• to get each student functioning at work on a higher level 

• to get them ready for further training 

• to make them greater assets to the company 

• to communicate more/better with each other and with management 
(ask/answer questions, read procedures) 

• to improve critical thinking skills 

Evidence of goal attainment 

• students speaking more in class, asking more questions (or different 
kinds/order of questions) 

• students articulating their needs more 

• students participating in/contributing to discussions 

• students organizing ideas in writing 

• students mastering use of grammatical structures 

• students improving reading comprehension (reading more difficult 
texts) 

• increased confidence: less hesitancy in speaking, more eye contact, 
greater willingness to talk 

For the ESL teachers, the funding source and the workplace environment 
have affected their teaching. They find the focus to be more narrow, with 
company needs having priority; however, they also articulated their own, 
broader goals. One teacher pointed out that the focus is on English as a 
vehicle for communication rather than as isolated, abstract, and academic. 
Examples of how cl ass work relates to the workplace are: pronunciation 
activities to improve students' comprehension of others and others' 
comprehension of the students; reading skills such as previewing, scanning, 
finding main ideas and verifying details (relates to reading work orders); 
asking and answering short questions (to address supervisors' complaints 
that students do not ask when they do not understand and do not respond to 
direct questions); job titles and job duties. On the other hand, another 
teacher focused her comments on "academic" skills, such as reading grade 
levels. Increase in grade level was her primary indicator of progress and 
success. 
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CONCLUSION 

The purpose of this report is to help CEI improve its workplace education 
services to organizations. This involved determining what went well in the 
last two years of Global 2000 and what could be improved. The data 
collected demonstrate that the classes which CEI provides are strong. 

There is a partnership at each site among key stakeholders, although there 
is variation from site to site regarding the extent to which all stakeholders 
contribute to goals and activities. All sites have a plan and goals statement; 
at this point, these plans should be reviewed. I recommend revising if 
necessary to make the goal and indicators clearer and easily measurable. In 
addition, as indicated in Section 3, data collection instruments may need to 
be revised. However, as also indicated in Section 3, there is evidence of 
results at all sites, both with regard to impact on students and with regard 
to impact on the workplace. The greatest weakness of Global 2000 is in 
collecting and analyzing data, but if revisions are made as recommended 
throughout this report, this may be improved. Since one site has done a 
great job of collecting data (Fire Controls), they may be used as a model 
for other sites. 

The task of determining goal achievement depends on having clearly 
articulated and measurable goals. It also means developing procedures, 
such as regular focus groups, for collecting anecdotal information. Refer 
to the attachments for sample forms and procedures which may help when 
revisiting this process with the Employee Involvement Teams. A highly 
useful reference is Lynch (1996) Language Program Evaluation: theory 
and practice . Melbourne: Cambridge University Press. Also refer to the 
next issue of Adventures in Assessment , put out by World Education. 
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EIT FOCUS GROUP - DRAFT PROTOCOL 



Introduction 

CEI has asked me to come in and speak with you to help them determine in 
what ways the workplace education program have been successful so far, 
and in what ways their services and the program can be improved for next 
year. Besides meeting with you (the EIT), I will be meeting with some 
former participants and with some teachers. Everything you say will be 
kept confidential. I will only be reporting aggregate information/results. I 
appreciate your openness in speaking with me. 

(at end: Thank you for your time) 

General Information about the Team 



• How often do you meet? For how long each time? 

• Who facilitates the meetings? Is there a standard format? (what is a 
typical meeting like?) 

• What are the responsibilities of the team? What does the team do? (get 
specifics of what happens in a meetings and outside) Why? 

• Do you think a team is an effective governing body for the workplace 
education program? (explain how, why) 

• What challenges do you face in doing these tasks? (explain) 

• What would help make the team more effective in carrying out its 
responsibilities? 

• Are there other tasks you think the team could/should be involved in, 
but hasn't? What and why? 

Is the EIT what you expected? How? 

• How would you describe the relationship between CEI and the 
company? 
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Information on the Program 



• What are the program's philosophy and goals, to the extent articulated? 
How do these relate to the company's philosophy and goals? 

• What did you like/dislike about the last two years? 

• Did the program meet your expectations? How, how not? 

• Now that you've experienced the program for two years, have your 
expectations changed? How? 

• What do you see as the strengths of the program? 

• Are there any ways you think the program could be improved? 

• What does 'success' mean to you? What counts as 'evidence' of success? 

• What changes do you look for to see whether students are 'succeeding' 
or making progress in their English language skills? What kind of data 
do you collect? 

• How does the program connect with other training and development/ 
organizational change initiatives? 

• By the end of year 3, what would you like to see happen/change? 

• The company has the following goals for Global 2000 (remind them of 
goals and measures) 

• What is the relationship between the ESL program and these goals? 

• How does the ESL program help the company achieve these goals? 
(specific examples) How do you know? 

• Are there opportunities for workers to apply new skills gained through 
the program at work? (examples) 

• What evidence is there, if any, of results of worker participation in the 
education program on: 

The workers themselves (e.g. changes in job status, performance, 
understanding of co. information, etc.)? 

The workplace (e.g. quality of goods, etc.)? 
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Information on the Program 

• What classes did you take? When? 

• What did you like/dislike about the last two years? 

• What would you like to be different next year? 

• What would you like to be the same? 

• What are/were your goals? 

• Did the program meet your expectations? How, how not? 

• What do you see as the strengths of the program? 

• Are there any ways you think the program could be improved? 

• How do you know if you are learning or making progress? 

• Do you receive other training? 

• Are there opportunities for you to apply new skills gained through the 
program at work? (examples of when you are given or not given 
opportunities) 

• Does anyone monitor if and to what extent you apply new skills you 
learn in class at work? If yes, who and how? 

• Do you see any results/changes of participation in the program: 

On how you and/or others do your work or understand your role in 
the company? 

On yourself personally,in your family, and/or community? 

• Do you think there are other workers here who could benefit from 
classes but don't come? Why do you think they don't come (barriers to 
participation)? Is there a way we could help encourage them? 

• These are the company's goals for the program (tell them). 

Did you know these were their goals? 

Does the program/classes help achieve these goals? How, how not? 
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TEACHER PROTOCOL 



1 . What are your responsibilities with CEI? 

2. How long have you been at this site? What do you teach? 

3. Are there other teachers at your site? 

4. How much contact do you have with other teachers? Do you find this 
to be sufficient? 

5. Is the funding source (NWLP) or the fact that this is a workplace 
program important to you and to what/how you teach (i.e., does it 
make a difference)? In what ways? 

6. As you understand it, what are the program's philosophy and goals? 

7. What do you see as the strengths of the program? 

8. Are there ways in which you think it could be improved? explain 

9. What counts as evidence of progress to you? What changes do you 
look for to see whether students are making progress? (how do you 
note progress?) 
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APPENDIX B: DESCRIPTION OF THE EVALUATION 
PROCESS 



I. Background 

Over the course of eighteen months team members received a variety 
of instruction, support and technical assistance from the evaluation 
facilitators both on site and in state-wide meetings. There were four 
evaluation facilitators who worked with the local teams. The facilitators 
were: Bob Bozarjian, who was the Coordinator of the MWEI, NWLP 
Wave IV Program; Johan Uvin, who was the Assistant Coordinator; Kathe 
Kirkman, who was the Evaluation Specialist in the Adult and Community 
Learning Services Cluster; and Laura Sperazi, Director of Evaluation 
Research who was also the external evaluator. The facilitators took the 
teams through a twelve stage process which is summarized below. For a 
more complete description of the team evaluation process, refer to "Team 
Evaluation: A Guide for Workplace Education Programs" which is 
available through the ERIC Clearinghouse and from Paul Jurmo, Literacy 
Partnerships. (See "References and Resources" for more information.) 



II. Twelve Steps of Team Evaluation 

Step 1 ; The team is convened and introduced to the 
principles of team evaluation. The team meets for the first 
time, reviews relevant materials on team evaluation, and considers 
the challenges and benefits of working together on an evaluation 
project. 

Step 2: The team clarifies its expectations regarding 
evaluation. The team builds the foundation for its evaluation 
activities by answering the basic evaluation question: Who wants 
what information for what purpose? Differences in stakeholder 
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expectations are clarified. Differences between summative and 
formative evaluation questions are also clarified. 

Step 3: The team identifies the goals it wants to evaluate. 
Summative or outcome evaluation: the team identifies program 
goals with special attention to if/how goals differ across stakeholder 
groups, and chooses the goals it wants to evaluate. 

Step 4 : The team identifies appropriate indicators for its 
goals. The team answers the question "How will we know if we 
have met our goals?" by specifying the indicators (or "signs of 
success") for its goals. These indicators become the building blocks 
of the surveys, interviews, focus groups, and other information 
gathering procedures which the team will use to gather summative 
information. 

Step 5: The team clarifies which program components 
need to be in place in order to achieve the desired goals, 
and evaluates whether the program is operating according 
to its own standards of quality. Formative or process 
evaluation: the team evaluates how the program is conducted. It 
establishes "quality standards" for each program component, 
determines if current operations meet the standards needed to 
achieve its desired goals, and — if not — develops an action plan to 
bring those operations into "quality range." The team thus develops 
an action plan to improve program operations so that the likelihood 
of achieving desired goals is enhanced. Teams will revisit goals, 
indicators, needed program components, and action plans as needed. 

Step 6 : The team formulates an evaluation plan. The team 
considers key issues in evaluation design and then thinks through 
when, how, and from whom it will collect the summative (outcome) 
information it wants -- as well as any additional formative (process) 
information it wants which the previous exercise did not capture. 
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— ep The team designs and pilots instruments and 

procedures for collecting the desired information. The 
team chooses and/or designs data-gathering procedures. The team 
pilots these procedures as a basic check for reliability and validity. 

Step_8: The team implements its evaluation plan. After 
designing and piloting its data gathering procedures, the team carries 
out its evaluation plan. This includes inviting potential respondents 
to participate in the evaluation, creating the appropriate conditions 
for collecting information, scheduling the data collection, etc. 

Step_9: The team gathers and organizes its data. Designated 
team members gather and organize the data. 

Step_10: The team analyzes its data. Team members analyze 
the data. 

Step_JT: Team members prepare a strategy for reporting 
their findings. Team members consider the range of options for 
reporting significant findings and target their evaluation audiences. 

St§p_J2: Team members report their findings to targeted 
audiences and incorporate their findings and audience 
feedback into program planning and future evaluation 
strategies. The team reports its findings and uses what it learns in 
two ways: to inform strategic planning decisions and to clarify the 
next evaluation questions which it will answer. 
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GOALS 

OF GLOBAL 2000 





Pow. Gen. 


Analog Dev. 


AEC 


Fire Control 


Boston Sci. 


Improve 
productivity 
of workplace 


data needed? 


data needed? 




data needed? 


data needed? 


Empower 

participants: 

1) to advance 
in jobs 

2) to improve 
job 

performance 

3) to retain 
jobs 


data needed? 


data needed? 




data needed? 


data nneded? 


Demonstrate 

a model for 

successful 

workplace 

literacy 

programs in 

the 

manufactur- 
ing sector, 
through 
formation of 
integrated 
partnership 


partnership 

exists 


partnership 

exists 




partnership 

exists 


partnership 

exists 
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MEASURABLE OBJECTIVES AND INDICATORS 
OF GLOBAL 2000 



OBJECTIVES 


INDICATORS 


SITE 


Form an EIT 






Conduct a lit/skills 
audit of literacy, 
lang., math reqs of 22 
types of jobs 






Conduct needs 
analysis with sup. 
staff and employees 
to establish 
contextualized 
workplace specific 
curr. needs 

to determine means, 
measures for 
evaluating needs 






Conduct orientations 






recruit min. 647 
participants over 3 
yrs 






Conduct pre-math, 
rdg assessments 






Conduct ind. 
interviews w/647 






Provide EWL 
instruction for 216 






Provide math 
instruction for 1 96 






Provide rdg comp and 
writing for 1 96 






achieve participant 
completion rate of 
80% per 10-wk cycle 
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Achieve noticable 
improvement b/t pre 
and post writing 
samples 






Achieve 50% increase 
in math scores for 
80% of participants 






Achieve 1 grade level 
increase for 80% of 
participants on rdg 
test 






Conduct quantitative 
and qualitative evals 
of project - done by 
EIT 






Provide ADP to 54 






Provide computer 
aided instruction and 
tutoring to 
supplement other 
instruction 






record and monitor 
improvements in 
1 )productivity 

2) participant job 
performance 

3) job retention and 
advancement 
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COMPLETE LIST OF 
PROGRAM GOALS 



PRIORITIZING PROGRAM GOALS FOR EVALUATION 






Identify Indicators 

1. How will we know whether this goal has been achieved or not 

2. What will we take as evidence? 

3. What behaviors, attitudes, or events will show it's achieveme 
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Draft Guidelines for Curriculum Documents s/'o/ 

Curriculum Working Group 
Massachusetts Workplace Education Consortium 



For each curriculum document, please include: 

• Statement of your teaching philosophy/approach 

• Description of your teaching context (workplace, class level, size, ESL or ABE, etc.) 

• Description of your curriculum development process: 

- Explain how you have elicited and incorporated student input and needs 

- Explain how you have elicited and incorporated input and needs of other 

stakeholders 

- Explain how you elicited and incorporated workplace topics and materials 

- Explain methods you used to assess student progress, effectiveness of teaching, etc. 

• Description of the challenges you learned from, what you would do differently 

• Sample "chunk” of your teaching activities. Describe/include the following: 

- How the topic was chosen 

- Teaching/class goals 

- List of skills addressed 

- Time frame 

- Processes and activities used 

- Materials used (please attach) 

- Assessment tools used, if any (please attach) 

- Examples of homework, if any (please attach) 

- Reflections on the lesson 

• List of topics covered in your cycle 

• Resource list of published materials (texts, photos, etc.) you found helpful/have used 



The Curriculum Working Group is made up of teacher representatives from each of the seven 
Consortium partnerships around the state. 
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REPRODUCTION BASIS 




This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the BRIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
docs not require a “Specific Document” Release form. 




I his document is f ederal ly-lundcd, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by BRIC without a signed Reproduction Release form 
(cither “Specific Document” or “Blanket”). 
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